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Abstract 

Creating virtual models of real spaces and objects is cumber- 
some and time consuming. This paper focuses on the prob- 
lem of geometric reconstruction from sparse data obtained 
from certain image-based modeling approaches. A number of 
elegant and simple-to-state problems arise concerning when 
the geometry can be reconstructed. We describe results and 
counterexamples, and list open problems. 

1 Introduction. 

While three-dimensional virtual models have long been 
used in industry for design, the increased speed and 
graphics capabilities of today's computers, higher band- 
width, and the popularity of virtual environments mean 
that virtual models are becoming ever easier to view, 
manipulate, and distribuite. This improved ease of use 
has spawned an increasing desire for better methods 
to create models, including models of real objects and 
spaces. 

At FXPAL, we are particularly interested in the use 
of virtual models in a factory setting [llj and in surveil- 
lance [151 126j . Common applications include training, 
immersive telepresence, military exercises, and design 
and testing of emergency response plans. Other appli- 
cations range from virtual tourism [281 127j and psychi- 
atric treatment for post-traumatic stress disorder [T4] . 
phobias 21j, and autism [3D]. Real estate offices are 
beginning to use three-dimensional models to support 
the creation of virtual tours [4 . Not only are marketing 
departments beginning to make models of their prod- 
ucts available to potential purchasers, but applications 
are springing up around these models. For example, 
MyDeco ISj enables users to create models of a three- 
dimensional space, place models of real furniture and 
other home accessories that are available for purchase 
in the virtual space, and then buy any of these products 
directly from the site. Virtual worlds such as Second 
Life [7] are filled with more or less realistic models of 
real places and objects. Google Earth 3^ now includes 
three-dimensional models of various buildings. 

Unfortunately, creating virtual models of real ob- 
jects and spaces remains cumbersome and time con- 



Figure 1: Model of an IKEA Bookcase cabinet gener- 
ated by the Panthcia system. 



suming. Current state of the art modeling is done by 
artists using interactive modeling tools, often supported 
by measurement and photographs of the real space. An 
ambitious long term research goal is to automatically 
construct such models from collected images; fully au- 
tomatic approaches are not yet possible. FXPAL's Pan- 
thcia system [ITj |25j enables users to create models by 
marking up the real world with pre-printed uniquely 
identifiable markers. Predefined meanings associated 
with the markers guide the system in creating mod- 
els. The position and outward pointing normal at each 
marker can be estimated from user-captured images or 
video of the marked-up space. Point-normal data, con- 
sisting of the position and outward pointing normal, can 
be obtained using other technologies such as range scan- 
ners. 

This paper focuses on the problem of reconstructing 
the geometry from the marker information. Our initial 
attempts at reconstruction used ad hoc reconstruction 
algorithms and markup placement strategies. When 
we tried to model a new space, we often needed to 
place additional markers, add meanings to the markup 
language, or revise the reconstruction algorithm to make 
it more powerful. This paper is the result of our work to 
place the geometric reconstruction aspect of our system 
on a firmer formal footing. 



2 Related Work. 

This section discusses two types of related work. First, 
we discuss related work in the area of image-based 
modeling. Then we survey previous work in polyhedral 
reconstruction from simple geometric data. 

Researchers such as [531 IMl US] work on non- 
marker-based methods for constructing models from im- 
ages. Their work advances progress on the hard prob- 
lem of deducing geometric structure from image fea- 
tures. Instead, we make the problem simpler by plac- 
ing markers that are easily detected and have meanings 
that greatly simplify the geometric deduction. Further- 
more, a marker-based approach enables users to specify 
which parts of the scene are important. In this way, 
Pantheia handles clutter removal and certain occlusion 
issues easily, since it renders what the markers indicate 
rather than what is seen. 

From a large number of photographs of a place 
or object, visual features, such as SIFT features |19[ . 
can be extracted and rendered as point cloud models 
[2H1 [13 • More generally, the area of 'point based 
graphics' provides methods for representing surfaces by 
point data, without requiring other graphics primitives 
such as meshes [16 . These methods have been used 
as primitives for modeling tools [9 . Amenta et al. [TOj 
describe the point based notion of 'surfels' which are 
points and normals. Our markers specify one dimension 
over a surfel: the orientation of the marker within 
its plane. Generally, point based methods use large 
numbers of surface points, and aim to produce smooth 
surfaces. Our system aims to produce polyhedral 
models with low polygon count from sparse geometric 
data. 

There is a rich history of work on reconstructing 
polyhedra from partial descriptions. See Lucier [2Dj for 
a survey. There is little work, however, on reconstruct- 
ing polyhedra from sparse point-plane or point-normal 
data, let alone more complex metadata. Biedl et al. dis- 
cuss several polygon reconstruction problems based on 
point- normal data and related data |12j . Their recon- 
struction results are limited to two dimensions. 

3 Overview of the Pantheia System. 

For markers, the Pantheia system uses the two- 
dimensional two dimensional barcode style markers of 
ARToolKitPlus [U (see Figure ^. Pantheia ^ [13 
takes as input a set of images, identifies the markers 
in each image, and determines the relative pose of the 
marker to camera for each marker in each image. If 
the pose of a marker in the world is known then the 
pose of the camera for each image can be determined. 
Conversely, if the pose of a camera is known then the 
pose of any marker identified in that image can be cal- 
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Figure 2: Examples of visual markup markers 

culated. From a set of images that meets a few simple 
conditions, the pose of every marker and the pose of the 
camera for every image can be estimated. Pantheia ob- 
tains good estimates using ARToolKitPlus [T] together 
with the sparse bundle adjustment package SBA (TH] 
that globally optimizes the pose estimates for all mark- 
ers and images. 

Pantheia creates models using a markup language 
|17j that includes elements specifying appearance char- 
acteristics, interactive elements, and geometric proper- 
ties of the scene. This paper concerns only the geomet- 
ric reconstruction aspect of the system supported by the 
following marker language elements: 

• planar (plane, wall, ceiling, floor, door), 

• shape (parametric shape, extrusion), 

• modifier (edge, corner). 

This paper reports on results of our ongoing effort 
to define a simple yet powerful markup language and to 
develop robust reconstruction algorithms that together 
support the creation of a large class of virtual mod- 
els. The output of our system can be thought of as the 
description of a virtual model and its dynamic capabili- 
ties in terms of a language such as COLLADA [2j , which 
supports the expression of physics, or of a language such 
as VRML W together with physics specifications for a 
physics engine such as ODE ^ in order to support in- 
teractive elements. Pantheia currently saves models as 
VRML together with metadata files that specify rela- 
tions between named parts of the VRML scene graph. 

The user is encouraged, when placing markers on 
the same plane, to use markers that have the same 
plane ID. Furthermore, planes that are close to aligned 
are forced to align by averaging their normal vectors. 




tions. Section 4.2 formally defines a markup description 
and related concepts. Section [473| defines various types 
of marker data considered in this paper. Section |44] dis- 
cusses the markup strategies considered in this paper. 



Figure 3: Pantheia generated model of a room 



Finally, there is an option that takes all planes close 
to being aligned with one of the coordinate planes and 
snaps them to being precisely parallel to the coordinate 
plane. These features all help with the robustness of 
the estimation. Accuracy results for an early version 
of the system are reported in [T7]. More details on the 
Pantheia system can be found in both [T71 US] . 

As Pantheia's designers, we can choose which 
markup strategies to suggest to users, which marker 
meanings to make available, and which reconstruction 
algorithm to use. Our aim is to design markup strate- 
gies that, for a broad class of models, specify a unique 
model, are easy for users to understand and carry out, 
and have an efficient reconstruction algorithm. 

4 Formal lYamework. 

This section contains definitions used through the pa- 
per, and sets up the formal framework in which we dis- 
cuss markup strategies and reconstruction algorithms. 
The framework is somewhat abstract in that we as- 
sume that the position and orientation of all markers 
is known. When marker poses are obtained using vision 
techniques, some markers may not be visible to a cam- 
era. For those markers, a pose cannot be obtained. We 
do not handle these cases, nor do we discuss robustness 
questions. Both are interesting areas for future work. 



Section 4.1 begins with some basic geometric defini- 



Section 4.5 covers elaboration techniques for construct- 
ing more complex polyhedra from simpler ones. 

4.1 Basic geometric definitions. We begin with 
some basic definitions. This paper is primarily con- 
cerned with constructing polyhedra from marker data. 
Surprisingly, there are a number of variations as to how 
the terms "polygon" and "polyhedron" are defined. For 
clarity, we state the definitions we use in this paper. 

Definition 4.1. A polygon is a closed, connected, two- 
dimensional region of a plane whose boundary consists 
of a finite set of segments, ei,e2, ...e„, called edges with 
endpoints, vi^V2,---Vm called vertices. Furthermore, 
every vertex Vi is the endpoint of two edges, each edge 
ends in two vertices, and two edges can intersect only 
in a vertex. 

Definition 4.2. A polyhedron, pi. polyhedra, is a 
closed, connected, three-dimensional volume of three- 
dimensional Euclidean space whose boundary consists 
of a finite set of polygons called faces such that every 
edge of every polygon is shared with exactly one other 
polygon, two faces may intersect only at edges or ver- 
tices shared by the two polygons, and the faces that 
share a vertex can be ordered so that face fi shares an 
edge with /i+i (mod q, the number of such faces). 

This definition allows polyhedra to have holes. Our 
theorems hold for polyhedra of arbitrary genus. 

Definition 4.3. A region is convex if, for every pair of 
points in the region, the line segment connecting those 
points is entirely contained in the region. 

Definition 4.4. A polyhedron is orthogonal if all of its 
faces are parallel to one of the three coordinate planes. 
A set of polyhedra is orthogonal if all of the polyhedra 
in the set are orthogonal. 

4.2 Markup definitions. We model a marker as a 
numeric identifier together with metadata. Minimally 
the metadata includes the position of each marker. 
The metadata may include other information about 
the marker or its placement, such as the outward 
pointing normal, or meanings from the markup language 
associated with sets of markers. We assume that all 
marker information is known - that all markers have 
been seen and robustly estimated. 

Definition 4.5. A markup description of a model is a 
set Im of marker indices, together with the metadata 
associated with those markers. 



Let X be the class of models under consideration. 

Definition 4.6. A markup placement strategy fp is a 
mapping fp-.X-^V from the class of models X to the 
space of markup descriptions V. 

A markup strategy may or may not be determinis- 
tic. For example, a deterministic strategy might be, "for 
each face of the model, place a marker on the face at the 
centroid of its vertices" . A nondeterministic strategy 
might be, "For each face of the model, place a marker 
somewhere on the face" . 

Definition 4.7. A markup reconstruction rule fn is a 
mapping from the class of markup descriptions V to the 
class of models X. A markup reconstruction algorithm 
is a procedure which implements a reconstruction rule. 

Definition 4.8. A markup system is a markup place- 
ment strategy fp together with a reconstruction rule 
fn. A markup system is complete and faithful if for 
every mG X, fpifpim)) = to. 

4.3 Marker data and metadata types. This pa- 
per considers three main types of marker data. 

Definition 4.9. Point marker data consists of the 
position of the center of the marker, or the position of 
a point specified by the metadata relative to the center 
of the marker. The corner marker shown in Figure [2] is 
an example of a marker that specifies a position that is 
not the center of the marker. 

Definition 4.10. Point-plane metadata consists oi the 
position of the marker and the plane in which it lies. 

Definition 4.11. Point-normal metadata consists of 
the position and outward pointing normal at each 
marker. 

To each of these basic types, various levels of 
additional metadata can be added. Useful types of 
additional metadata include IDs that indicate that all 
markers with that ID share a property such as being 
on the same face or defining the same polyhedron, 
orderings of a set of markers that indicate, for example, 
the order in which to traverse the vertices of a face, and 
relationships, such as two faces share an edge. 

4.4 Some Markup Strategies. 

Definition 4.12. The marker-per-face markup strat- 
egy is any placement of at least one marker on each 
face. 



Most of this paper will discuss reconstruction from 
a marker-per-face strategy with point-plane or point- 
normal metadata and possibly additional metadata. As 
mentioned in Section [2j much more common in the 
literature are discussions of reconstruction from a vertex 
markup strategy in which every vertex is marked. We 
are less interested in such strategies than marker-per- 
face strategies because they require more precision in 
placement on the part of a user and require markers to 
be placed in places that may be out of reach or even 



hidden. In Section 5.3 we discuss a few results related 



to vertex or edge markup strategies. 

4.5 Elaboration. Elaboration is a way to create a 
more complex polyhedron from a base polyhedron by 
gluing a polyhedron to a face or removing a polyhedron 
aligned with the face from the base polyhedron. Ex- 
trusions and intrusions are special cases of elaboration. 
The reconstruction results related to elaboration focus 
on reconstruction of polyhedra that can be obtained by 
elaborating a convex polyhedra with extrusions and in- 
trusions of convex polygons. 

Definition 4.13. Orthogonal Polygonal Extrusion or 
just Extrusion: A polygon in the interior of a face of 
the base polyhedron is extruded outward, perpendicular 
to the face, a constant amount. 

Definition 4.14. Orthogonal Polygonal Intrusion or 
just Intrusion: A polygon in the interior of a face 
is pushed inward, perpendicular to the face, a constant 
amount. 

We will also consider polyhedra obtained by taking 
the union of separately defined but intersecting polyhe- 
dra, including ones obtained by gluing along a shared 
face. 

5 Reconstruction Results. 

Our goal is to understand under what circumstances 
unambiguous reconstruction is possible from complete 
knowledge of all marker poses and metadata. At the 
same time we would like the user's task to be as simple 
as possible. We have two starting points, one easy for 
the user but supporting reconstruction of only a small 
class of polyhedra, the other general but burdensome on 
the user. 

Theorem 5.1. (Convex polyhedron) A marker- 
per-face strategy with point-plane metadata is sufficient 
to uniquely specify an arbitrary convex polyhedron. 

Theorem 5.2. (Dense markup) For any scene con- 
taining finitely many polyhedra, if markers are placed 



Figure 5: A configuration of markers tliat supports two different, but topoiogicaily equivalent, interpretations 
under tlie "single marker per face" markup system. 



Figure 4: A configuration of markers ambiguous under 
tlie "single marker per face" markup strategy. Two 
polyhedral interpretations of this marker set are shown. 



sufficiently densely, then the scene can be reconstructed. 
More precisely, for any scene containing a finite num- 
ber of polyhedra, there is an e such that if every point is 
within e of a marker, then the scene can be unambigu- 
ously reconstructed. 

The problem with the marker-per-face approach is 
that Theorem 15.31 and Theorem 15.41 show that both 
convexity, and the requirement that the scene contains 
only one convex polyhedron, are necessary to guarantee 
unambiguous reconstruction. The two main concerns 
with respect to dense placement are that while there 
is a constructive means to determine e, as the proof 
shows, it is not easy for a user to visually determine 
what density suffices, and it requires users to place many 
more markers than are necessary, including in places 
that may be hard to reach or even hidden. 

The rest of this section pursues ways of enabling 
users to specify broader classes of models while keeping 
the burden on the user low in terms of the number 



of markers a user must place, the complexity of the 
instructions, and the precision with which the user must 
place the markers. 

Short proofs follow or preface the statement of the 
results. Longer proofs are contained in Appendix|Xj For 
example, the proofs of both Theorem |5 . 1 1 and Theorem 



5.2 are contained in Appendix [A| 



5.1 Point-plane marker-per-face results. Theo- 
rem |5.1| states that a marker-per-face strategy with 
point-plane metadata suffices to uniquely reconstruct 
a convex polyhedron. Convexity is required, however, 
to avoid ambiguity. 

Theorem 5.3. A marker-per-face strategy with point- 
plane metadata is not sufficient to uniquely specify a 
non- convex polyhedron. 

Proof. Figure [4] provides a simple counterexample. This 
example forms the basis for a more complex arrange- 
ment of markers that are consistent with two differ- 
ent polyhedra, shown in Figure [5] that have the same 
topology. Both examples can be extended to three di- 
mensions by placing two markers on both an upper and 
lower face in the positions marked by the gray squares. 

Similarly, that there is only one polyhedron in the 
scene is required to avoid ambiguity. 

Theorem 5.4. A marker-per-face strategy with point- 
plane metadata is not sufficient to uniquely specify 
multiple convex polyhedra. 

Proof. Figure [6] 



Theorem |5.3| and Theorem |5.4| hold for point-normal 
metadata as well. 

The ambiguities in the case of multiple convex 
polyhedra is easily overcome by adding a polyhedron ID 
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Figure 6: A configuration of markers tliat is ambiguous for tlie 'collection of non-intersecting convex polygons' 
model class with the 'single marker per edge' markup rule. 




to the metadata so that all markers placed on the same 
polyhedron have the same polyhedron ID and markers 
placed on different polyhedra have different polyhedron 
IDs. The user's task is now slightly more burdensome 
in that the user must keep track of different sets of 
markers for different polyhedra in the scene, but that 
is a relatively light additional burden. 

Theorem 5.5. A marker-per-face strategy with point- 
plane and polyhedron ID metadata suffices to uniquely 
specify multiple convex polyhedra. 

We obtain more general results by considering elab- 
orations of simple polyhedra. 

5.2 Markup Containing Elaboration Informa- 
tion. We begin by stating the most general elaboration 
result we have and then discuss situations in which less 
elaborate markup suffices. 

Theorem 5.6. (Hierarchical elaboration) Any 
polyhedral scene that can be constructed by starting with 
a finite set of convex polyhedra and iteratively elaborat- 
ing faces with orthogonal convex polygonal extrusions 
and intrusions can be unambiguously reconstructed from 
a marker-per-face strategy with point-plane metadata 
together with the following additional metadata that 
precisely specifies an elaboration hierarchy: each of the 
polyhedra from which the scene will be built must have 
its own polyhedron ID, and each of its faces identified 
as faces of the initial set of polyhedra from which the 
construction is made, and each elaboration has its own 
ID and the markers for an elaboration indicate on 
which base face the elaboration is made. 

By including additional ordering information in the 
metadata, extrusions and intrusions from nonconvex 
polygons can also be handled. 



Theorem 5.7. Theorem \5.6[ can be extended to the 
case of orthogonal nonconvex polygonal intrusions and 
extrusions if for each elaboration, the markers on 
the faces perpendicular to the base of the elaboration 
include additional metadata providing an ordering, say 
clockwise, of these faces. 

A few remarks are in order. 

• Separate IDs are required for each elaboration in 
order to handle cases in which the elaboration and 
markup resemble that shown in Figure [6] 

• A given scene can often be constructed by different 
series of elaborations. This nonuniqueness of the 
hierarchical elaboration supports different users' 
conceptualization of how a scene can be built, but 
is also potentially a source of confusion when a user 
considers how best to mark up a scene. 

• Because the different polyhedra each have their own 
ID, this proof applies to scenes in which the differ- 
ent polyhedra intersect. The model can be rendered 
as intersecting polyhedra, but for efficiency reasons 
it may be advisable to simplify the resulting model 
by removing all regions internal to the final collec- 
tion of polyhedra. While this capability extends 
the class of scenes we can reconstruct, it can add 
to the confusion of a user as to how best to markup 
a scene. One subcase of intersecting polyhedra is 
polyhedra that can be obtained from simpler poly- 
hedra by gluing along their boundaries. 

Many real world objects and spaces are orthogonal 
elaborations of convex polyhedra. An even larger class 
of objects is well approximated by such polyhedra. 
When the set of extrusions and intrusions in each face 
are derived from a set of non-intersecting rectangles that 
are axis aligned for some choice of two perpendicular 



axes, the separate elaborations do not need different 
elaboration IDs. 

Theorem 5.8. Let V be a set of polyhedra obtained 
from a set of convex polyhedron by intrusions and ex- 
trusions, where all of the intrusions and extrusions in 
each face are non-intersecting, orthogonal rectangular 
intrusions. This set of polyhedra can be uniquely re- 
constructed from a marker-per-face strategy with point- 
normal metadata in which 

• all markers defining a single polyhedron have an 
associated polyhedron ID, 

• the markers specify the base faces of each polyhe- 
dron, and 

• all markers indicating an elaboration specify the 
base face in which the elaboration is done. 

Even the class of real world objects and spaces that 
can be obtained simply by orthogonal intrusions of a 
box is reasonably large. In this case, a simple marker- 
per-face strategy with point-normal metadata suffices 
without the need for additional metadata. 

Theorem 5.9. Let P be a polyhedron obtained from a 
box by a set of non-intersecting rectangular intrusions 
made in the interior of one face and aligned with the 
sides of this face. The polyhedron P can be uniquely re- 
constructed from a marker-per-face strategy with point- 
normal metadata. 

Theorem |5.9| can be strengthened to only need 
point-plane metadata, though the proof is more in- 
volved. 

Theorem 5.10. Let P be a polyhedron obtained from a 
box by a set of non-intersecting rectangular intrusions 
made in the interior of one face and aligned with the 
sides of this face. The polyhedron P can be uniquely re- 
constructed from a marker-per-face strategy with point- 
plane metadata. 

5.3 Other sorts of markup. As mentioned in Sec- 
tion[2] researchers have looked at polyhedral reconstruc- 
tion from vertex data. We are less interested in such re- 
construction results because they require placement of 
markers in hard to reach, or even hidden, places. Nev- 
ertheless we state a few results. 

O'Rourke gave an elegantly simple algorithm [22, 
that reconstructs orthogonal polygons from vertex data 
provided there are no degenerate vertices connecting 
two coUinear segments. This condition together with 
orthogonality means that from every vertex extends 



exactly one horizontal segment and exactly one vertical 
segment. The reconstruction is then obtained using 
a straightforward connect-the-dots algorithm: Consider 
each row of vertices. In each row, the first vertex must 
connect to the second, the third to the fourth, and so on. 
Connecting the vertices of each column in pairs in the 
analogous way, completes the argument. O'Rourke's 
algorithm can be extended to three-dimensions, again 
with a prohibition against no 180° vertices. This 
prohibition is more stringent in the three-dimensional 
case in that it rules out shapes we might like to cover 
such as one brick lying perpendicularly across another. 

Theorem 5.11. An orthogonal polyhedron in which all 
vertices do not connect any two collinear segments can 
be uniquely and quickly reconstructed via a connect-the- 
dots algorithm. 

Reconstruction of a much more general class of 
polyhedra is possible from vertex data together with 
some additional metadata. 

Theorem 5.12. Any polyhedral scene in which every 
face is convex can be reconstructed from a vertex markup 
with metadata that includes the face ID of all faces 
meeting at that vertex. 

Proof. Each face can be reconstructed by taking the 
convex hull of all of the vertices that are marked as 
associated with that face. 

General polyhedra can be reconstructed if the meta- 
data also includes ordering information. 

Theorem 5.13. Any polyhedral scene can be recon- 
structed from vertex markup if the metadata includes 
not only the face ID but also ordering information that 
specifies the clockwise order in which the vertices would 
be traversed when walking around the boundary of the 
face. 

Proof. The boundary of each face can be reconstructed 
by placing a segment between each vertex and the next 
one in the ordering. 

6 Future Work. 

We have explored a number of markup strategies with 
various types of metadata in search of markup systems 
that are are easy for users to carry out and that un- 
ambiguously reconstruct an attractive class of models. 
We are in the process of improving our system on the 
basis of these results. We look forward to working with 
artists to test our system, and to learn which strategies 
are most intuitive and effective for such users. 



A number of open research questions remain. We 
did not consider robustness issues. When the marker 
pose is obtained from camera estimates, there are in- 
teresting questions as to the best markup strategies in 
the face of inaccurate estimates or faihng to detect one 
or more markers altogether. Robustness in hght of user 
errors in entering metadata is also interesting. A num- 
ber of our results are not sharp, such as Theorem |5.6[ in 
that polyhedra not covered by the theorem can be recon- 
structed from the marker strategy and metadata. The 
question is whether the statement can be extended to a 
well-defined class of polyhedra. Similarly, in some cases, 
polyhedral scenes could be reconstructed with less meta- 
data. With care, some classes of polyhedra obtained 
by extensions from polygons with boundaries intersect- 
ing the boundary of the face could be covered. Con- 
sideration of non-orthogonal extrusions may be fruit- 
ful. We reconstructed a few non-polyhedral shapes such 
as cylinders. Extending these results to more general 
parametrized shapes would yield a much richer class of 
scenes that can be reconstructed. We primarily explored 
a marker-per-face strategy leaving open the question of 
whether a few markers placed more strategically would 
enable unambiguous reconstruction of a more general 
class of polyhedra. In general, we continue to search 
for practical markup strategies that yield unambiguous 
reconstruction of scenes from a sparse set of markers. 

References 

[1] ARToolkit. 

http : / / www. hit 1 .Washington, edu/artoolkit / . 

[2] COLLADA. http://www.collada.org. 

[3] Google earth, http://earth.google.com/. 

[4] Heartwood studios, http://www.hwd3d.com/. 

[5] Mydeco. http://mydeco.com/rooms/planner/. 

[6] Open Dynamics Engine, http://www.ode.org/. 

[7] Second life, http://www.secondlife.com/. 

[8] VRML. http://www.w3.org/MarkUp/VRML/. 

[9] Pointset3D. 

http://graphics.ethz.ch/pointshop3d/. 
[10] N. Amenta and Y. Kil. Defining point-set surfaces. In 

SIGGRAPH 2004, pages 264-270, 2004. 
[11] M. Back and T. Childs. High-tech chocolate: explor- 
ing 3D and mobile applications for factories. In SIG- 
GRAPH '09, pages 1-1, New York, NY, USA, 2009. 
ACM. 

[12] T. Biedl, S. Durocher, and J. Snoeyink. Reconstructing 
polygons from scanner data. In 18th Fall Workshop on 
Gomputational Geometry, 2008. 

[13] F. Dellaert, S. Seitz, C. Thorpe, and S. Thrun. Struc- 
ture from motion without correspondence. In Proceed- 
ings of CVPR'OO, 2000. 

[14] J. Difede and H. G. Hoffman. Virtual reahty exposure 
therapy for world trade center post-traumatic stress 



disorder: A case report. CyberPsychology and Behav- 
ior, 5(6):529-535, 2002. 

[15] A. Girgensohn, D. Kimber, J. Vaughan, T. Yang, 
F. Shipman, T. Turner, E. G. Riefi'el, L.Wilcox, 
F. Chen, and T. Dunnigan. DOTS: Support for ef- 
fective video surveillance. In ACM Multimedia 2007, 
pages 423 - 432, 2007. 

[16] M. Gross and H. Pfister. Pomt-based Graphics. Else- 
vier, 2007. 

[17] D. Kimber, C. Chen, E. Rieffel, J. Shingu, and 
J. Vaughan. Marking up a world: Visual markup for 
creating and manipulating virtual models. In Proceed- 
ings of ImmerscomOQ, 2009. 

[18] M. Lourakis and A. Argyros. sba: A generic 
sparse bundle adjustment C/C-\-+ package 
based on the Levenberg-Marquardt algorithm, 
http : / / www. ics .forth. gr / lourakis /sba/ . 

[19] D. G. Lowe. Object recognition from local scale- 
invariant features. In IGCV '99: Proceedings of the 
International Conference on Computer Vision-Volume 
2, page 1150, 1999. 

[20] B. Lucier. Unfolding and reconstructing polyhedra. 
http://uwspace.uwaterloo.ca/handle/10012/1037, 
2006. 

[21] K. Moore, B. Wiederhold, M. Wiederhold, and G. Riva. 
Panic and agoraphobia in a virtual world. Cyberpsy- 
chology and Behavior, 5(3): 197-202, 2006. 

[22] J. O'Rourke. Uniqueness of orthogonal connect-the- 
dots. Computational Morphology, pages 99-104, 1988. 

[23] M. PoUefeys. Visual 3D modeling from images, 
http : / / www. cs . unc.edu / marc /tutorial. 

[24] M. Pollefeys and L. V. Cool. Visual modeling: from 
images to images. Journal of Visualization and Com- 
puter Animation, 13:199-209, 2002. 

[25] E. Rieffel, D. Kimber, J. Vaughan, S. Gattepally, and 
J. Shingu. Interactive models from images of a static 
scene. In CGVR 2009, 2009. 

[26] E. G. Rieffel, A. Girgensohn, D. Kimber, T. Chen, and 
Q. Liu. Geometric tools for multicamera surveillance 
systems. In ICDSC 2007, pages 132 - 139, 2004. 

[27] W. Shao and D. Terzopoulos. Autonomous pedestri- 
ans. Graph. Models, 69(5-6) :246-274, 2007. 

[28] N. Suavely, S. M. Seitz, and R. Szeliski. Photo tourism: 
exploring photo collections in 3D. In SIGGRAPH '06: 
ACM SIGGRAPH 2006 Papers, pages 835-846, 2006. 

[29] N. Suavely, S. M. Seitz, and R. Szeliski. Modeling the 
world from internet photo collections. International 
Journal of Computer Vision, 80(2):189-210, 2008. 

[30] D. Strickland. Virtual reality for the treatment of 
autism. Studies m Health Technology and Informatics, 
44:81 - 86, 1997. 

A Proofs for results in Section [5j 

This appendix contains proofs of results in Section [5] It 
begins with proofs of Theorem |5.1| and Theorem |5^| 

To prove Theorem |5.1[ we make use of the following 
Lemma and Corollary. 



Lemma A.l. A convex polyhedron P lies entirely on 
one side of the plane defined by any of its faces. 

Proof. By contradiction. Suppose there are points of 
the polyhedron P on both sides of the plane R defined 
by a face F. We show that such a polyhedron cannot 
be convex. Let p be a point on face F. The face 
bounds the polyhedron on one side, so there exists an 
e > such that there are no points within e of p in 
one of the open half-spaces H defined by R. Since by 
hypothesis the polyhedron contains points on both sides 
of the plane R, let pi be a point in the polyhedron 
contained in the half-space H. Since the polyhedron is 
convex by hypothesis, the entire line segment between 
p and pi must be contained in the polyhedron. This 
line segment contains points within e of p. We have 
reached a contradiction, thus our supposition that there 
are points of P on both sides of F must be false. 



Theorem A.l. (Corollary to Lemma A.l) In 



convex polyhedron, the intersection of any plane R 
defined by a face F of a convex polyhedron with any 
other face F' of the polyhedron must be contained 
entirely within an edge of F' . 

Proof. If the plane R intersected the face F' anywhere 
other than in an edge of F' , the face F' would contain 
points on both sides of the plane i?, but that is 



impossible by Lemma A.l 



Proof. [Proof of Theorem 5.1 Convex Polyhedron] For 
each marker, determine the half-space which contains 
all the other markers, as per Lemma A.l Take the 



intersection of all these half-spaces. The intersection of 
convex sets is convex, and half-spaces are convex, so this 
intersection is convex. 

We want to prove that this convex polyhedron P/ is 
the same as the original one, Pqj that was marked. To do 
so, it suffices to show that no other convex polyhedron 
has faces with the same set of bounding planes. By 
Lemma I A. 1[ any convex polyhedron lies entirely on one 
side of the plane defined by any of its faces. By this 
Lemma, any other convex polyhedron defined by these 
faces must be contained in P/. To prove the converse, 
suppose Pq were strictly contained in P/. In this case 
its face set must be different. Because the face planes 
of Pi and Pq are identical, and Pq C P/ , a face Pq of Pq 
must be strictly contained a face P/ of P/. Let Pq be 
an edge of Pq that is not an edge of P/; in particular, 
Pq is a line segment in the interior of P/. The edge Pq 
must arise as the intersection of two faces of Pq, so must 
be contained in the intersection of two face planes Rj 
and Rj of Pq, one for the face P/ and one for another 
face, Pj, of Pq. The two planes Ri and Rj are also face 



planes for P/. By construction Rj intersects P/ in its 
interior, but that contradicts Corollary |A.1[ Thus Pq 
and P/ must be the same polyhedron. 

We now turn to proving Theorem 1 5. 2 1 



Proof. [Proof of Theorem 5.2 Dense Markup.] The 



markers define a finite number of planes. Partition each 
plane into regions along lines of intersection with all 
other planes. Remove from consideration all infinite 
regions. Each of the remaining regions contains a disk 
of positive radius for some radius e. Take the minimum 
Cmm of these positive radii over all of the regions. 
Since the set of regions is a finite set, this minimum 
exists and is positive. Unambiguous reconstruction of 
the scene can be done from any marker strategy that 
places at least one marker within distance d < tmin of 
every point on every face in the polyhedral scene. The 
reconstruction algorithm simply keeps every region in 
which there is a marker and discards the rest. 

A. 0.1 Proofs for Elaborations on a Convex 
Polyhedron. This section contains proofs for the re- 
sults stated in Section [sISl 



Proof. [Proof of Theorem 5.6 ] The marker metadata 



specified in Theorem |5.6| specified the entire hierarchy 
of elaborations by specifying to which face each elab- 
oration is made. The marker metadata tells us which 
markers lie on faces belonging to the set of initial poly- 
hedra on which the hierarchical construction is made. 
Start by identifying these markers. Because each initial 
polyhedron has its own ID specified by the marker meta- 
data, we can reconstruct the initial set of polyhedra by 
Theorem 15.11 

For each face on this initial set of polyhedra, identify 
any elaborations made to this face as indicated by the 
marker metadata. Each elaboration has its own ID, so 
to construct the elaboration, identify all markers with 
that ID that are perpendicular to the face. Projecting 
the markers onto the face defines a polygon. If there 
are any markers with this elaboration ID parallel to 
the face, they determine the depth of the extrusion 
or intrusion. If no such marker exists, the elaboration 
must be an intrusion and must go all the way through 
the polyhedron. Repeat this process for all faces in the 
initial set of polyhedra. The order in which each face is 
considered is irrelevant. This process is then repeated 
for the next level of the hierarchy: each face just created 
is considered in turn and any elaborations to those faces 
are determined. This process is continues for all levels in 
the elaboration hierarchy until there are all elaborations 
have been incorporated. 



We now prove some two-dimensional results to 
support the proofs of Theorem |5.9| and Theorem |5.10| 
We first prove a uniqueness results and then give an 
algorithm that performs the reconstruction. The proofs 
and algorithms make use of the following concepts. 

Definition A.l. A set of polygons is consistent with a 
set of markers if every marker is on (and aligned with) 
an edge of one of the polygons. 

Definition A. 2. A consistent set of polygons is fully 
consistent with a set of markers if in addition every edge 
in the set of polygons has at least one marker on it (and 
aligned with it). 

Definition A. 3. A marker is considered in the inte- 
rior of a polygon if its center is in the interior of the 
polygon, or it is on an edge of the polygon but not 
aligned with it. Markers on an edge, and aligned with 
it, are not considered in the interior. 

Consider the case of orthogonal non-intersecting 
convex polygons; in other words, we are considering a 
set of axis-aligned rectangles. Suppose this collection of 
rectangles is marked up using a marker-per-face strategy 
with point normal data. The markers can be divided 
into four classes depending on the direction of their 
outward facing normal, £, TZ, T, and B, where the 
normals for the markers in C point to the left, to the 
right in TZ, to the top in T, to the bottom in B. 

The rest of this section shows that a set of non- 
intersecting orthogonal rectangles can be uniquely re- 
constructed from any marker-per-face strategy with 
point- normal data. The idea behind the proof is that, in 
such a markup of an arbitrary collection of axis aligned 
rectangles, if we can find one rectangle P that belongs 
to a set of rectangles that is fully consistent with the 
marker set, and if we can show that every fully consis- 
tent set of rectangles contains this rectangle, by induc- 
tion there is only one set of rectangles fully consistent 
with the markers. We begin with a lemma that finds 
such a rectangle P. 

Lemma A. 2. Let Ai he an arbitrary set of markers 
placed according to a marker-per-face strategy with asso- 
ciated point-normal data on an arbitrary set S of non- 
intersecting, orthogonal rectangles. There is a unique 
rectangle P consistent with the left-most (and top-most 
among these if there is a tie) marker Ml G C that is 
contained in all sets of rectangles that are fully consis- 
tent with this set of markers. Moreover, the rectangle 
P is the only rectangle consistent with marker Af^ that 
has markers on and aligned with each of its edges and 
no markers in its interior. 



Proof. Suppose Ml lies on a rectangle P that is a 
member of a fully consistent set of non-intersecting 
rectangles S with respect to M, and that Ml also lies 
on a rectangle P' that is a member of another fully 
consistent set of non- intersecting rectangles S' . Let M^ 
be the topmost marker in TZ on the right side of of P, 
and Mt and Mb be the left-most markers in T and B 
on the top and bottom, respectively, of P. (See Figure 
[7j) Similarly, let be the topmost marker on the 
right side of of P' , and M!p and M'^ be the left-most 
markers on the top and bottom, respectively, of P' . 
Given the way we chose Ml, it must be the top-most 
marker on the left side of both P and P'. Furthermore, 
without loss of generality, we may assume that marker 
M!p is either to the right of Mr, or and Mt have 
the same horizontal component. (Otherwise, switch P 
and P'.) The marker M^ must not be higher than Mt 
since otherwise Mt would be in P', which is impossible 
since P' is part of a consistent set of non-intersecting 
rectangles. Unless Mt and M!p are the same marker, 
they cannot be equal in height, because then Mt would 
be a top marker on P' to the left of M!p. We are left 
with two cases to consider: either M^ is to the right 
and lower than Mt, or Mt = M'j.. 

Consider the case in which M'j. is to the right and 
lower than Mt- The marker must be to the right of 
Mfl, since cannot be in P. And, by definition, 
is above Ml- There are two cases: either Mr is above 
Ml or Mr is below Ml, as illustrated in Figure [7| (If 
Ml and M^ were equal in height, the P' would contain 
Mb..) Suppose Mr is above Ml- Then P' must be 
below Mr. But now Mn cannot be part of any rectangle 
in a set consistent with P'; there are no markers that 
could form the bottom of such a rectangle because there 
are no markers in B below and to the left of M^ and 
above P' since P contains no markers in its interior. 
Similarly, suppose Mr is below Ml- Then P' must be 
above Mr. But then Mr cannot be part of a rectangle 
in a set consistent with P' because there are no markers 
in T below P' and above and to the left of Mr. So the 
only alternative left open to us is that M'j, and Mt are 
the same marker. Similar arguments show that Mb and 
M'^ must be the same marker, and that Mr and M'j^ 
must be the same marker. Thus P is unique. 

Theorem A. 2. A set TZ of non-intersecting axis- 
aligned rectangles can be uniquely reconstructed from 
any set of markers M placed according to a marker- 
per-face strategy with point-normal data. 

Proof. We proceed by induction on the number of 
rectangles (which we do not assume is known). 

Let O be the bounding orthogonal rectangle, the 
smallest orthogonal rectangle that contains all the 
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Figure 7: Diagram supporting the proof of Lemma [A.2| 

markers. Each edge of O has at least one marker on 
it, and aligned with it, because by definition of O each 
edge of O must intersect the set of rectangles, and or- 
thogonality guarantees that this intersection must be 
in an edge, not just a point; we can find O from the 
markers by finding all markers that define a line which 
is a boundary of the set - a line is on the boundary if 
all markers are in only one of the half-planes defined by 
the line - and letting O be the intersection of all of these 
half planes. 

Case n = 1: If there is only one rectangle, O will 
be that rectangle. We know we are in this case if all of 
the markers are on and aligned with an edge of O. 

Case n — k: By Lemma [A. 2| any fully consistent 
set of rectangles must contain the rectangle P. Now 
exclude from any markers that are consistent with 
P. The remaining markers must be consistent with a 
set of fc — 1 rectangles. By induction, there is only one 
such set. 

Thus there is only one set of rectangles fully consis- 
tent with the set of markers M. 



The orthogonality assumption in Theorem |A.2| is 
necessary. 

Theorem A. 3. A set of non-intersecting convex poly- 
gons cannot always be reconstructed from a set of mark- 
ers that includes at least one marker per 



Proof. A counterexample is shown in Figure [6j 

A. 0.2 A reconstruction algorithm for Theorem 



for finding the unique set of rectangles fully consistent 
with a marker set. The algorithm proceeds by finding 
a rectangle P that is consistent with the top left-most 
marker M^. The rest of this subsection describes how 
that is done. Once it is done, all markers consistent 
with P are ignored, the new top left-most marker 
is considered, and a new rectangle is found. The 
algorithms continues until all markers have been taken 
care of. 

We set out to find a rectangle consistent with Ml 
that has markers on and aligned with each edge and 
no markers in its interior. Consider all markers in T 
that are to the right and above Ml; one of these must 
be able to form a rectangle with Ml that is part of 
a fully consistent set of rectangles for the marker set. 
Consider first the left-most of these markers and, if there 
are multiple left-most markers, take the top-most one of 
these. Call this marker M. 

Partition O into an irregular grid with lines defined 
by the markers in Ai. Consider the smallest rectangle 
consistent with markers Ml and M that is made up of 
grid segments. If this rectangle contains any markers in 
its interior, M cannot form a rectangle with Ml that is 
part of a fully consistent set, so go on to consider the 
next left-most marker. Eventually a marker MTcand is 
found such that the smallest grid-aligned rectangle R 
containing Ml and Mxcand does not have any markers 
in its interior. (There must be at least one, since by 
assumption, the markers mark the original set of non- 
intersecting rectangles, the set we are trying to find.) 

Check if there are markers on, and aligned with all 
sides of R. If there are, we take this rectangle to be 
P. Otherwise, expand the rectangle until this property 
holds as follows. Let S be the set of markers in B that 
are below and to the right of Ml- The set S cannot be 
empty since that would mean Ml is not on a rectangle 
that is part of a fully consistent set. Among the markers 
in S, consider the subset Sm of markers that do not have 
any markers in S directly above them or above them to 
their left. The left-most marker in S (and top-most, if 
there is more than one left-most) satisfies this criterion. 
Consider first the left-most (and top-most if there is a 
tie) of these markers, Mscand- If the smallest rectangle 
consistent with Mscand, Mxcand, and Ml has markers 
in its interior, go on to the next left-most marker in Sm- 
If there are no markers in the interior, find the left-most 
marker or markers in TZ and in the strip bounded on the 
top by Mxcand, on the bottom by Mscand, and to the 
right of Ml- If there is no such marker, then Mrcand, 
Ml, and Mscand, cannot form a marker consistent 
rectangle, so go on to consider the next candidate for 



Mf. 



If none of the possible candidates for Mbc 



A. 2, This section describes a constructive algorithm 



work, then try the next possible Mtc 



If there is 



no marker in the interior of the rectangle defined by 
MTcand, M^, and Mscandi find the left-most marker in 
TZ that is in the strip bounded on the top by Mxcand, 
on the bottom by MBcand, and on the left by Ml- If 
there is no such marker, then Mxcand, Ml, and MBcand, 
cannot form a marker consistent rectangle, so go on to 
consider the next candidate for MBcand- Again, if none 
of the candidates MBcand work, then consider the next 
candidate MTcand- Eventually some set works, since we 
know the markers form a fully consistent set for some 
set of rectangles. 

Definition A. 4. An infinite polygonal cylinder con- 
sists of a 3D region obtained by infinitely extending in 
both directions the edges of a polygon (that lies in 3D 
space) perpendicular to the face of the polygon. 

Definition A. 5. A polygonal half- cylinder consists of 
a 3D region obtained by infinitely extending in the edges 
of a polygon in one of the directions perpendicular to 
the face of the polygon. 



Proof- [Proof of Corollary 5.8 For each set of markers 



with the same polyhedron ID, construct the faces of 
the base polyhedron from the markers with metadata 
indicating that they are on one of the faces of the 
base polyhedron. For each base face F, consider all 
markers Sp that indicate that they are on faces made 
by elaboration of this face. Find all such markers that 
are perpendicular to this face, and project them onto 
the plane defined by this face. By hypothesis, the 
result defines a set of rectangles on the plane. Apply 
Theorem |A.2| to obtain the uniquely determined set of 
rectangles. Each rectangle corresponds to an intrusion 
or extrusion. Consider the infinite rectangular cylinder 
defined by each of the rectangles. If there is no marker 
contained in Sp in the interior of this cylinder, this 
rectangle defines an intrusion that goes all the way 
through the polyhedron. If there are markers in Sp 
inside that cylinder, they must all lie on the same plane, 
since rectangles defining the extrusions and intrusions 
do not intersect by hypothesis. This plane determines 
the extent of the extrusion or intrusion. The same 
construction can now be applied to all elaborations 
of faces that resulted from this level of elaboration, 
and can continue to be applied at each level until all 
elaborations have been handled. 



Proof- [Proof of Theorem 5.9 Consider all a;y-aligned 
markers. All of these markers lie on faces that are either 
boundaries of the original box or are faces obtained 
from making interior orthogonal rectangular intrusions 
in the same face. Because the intrusions are in the 
interior, all of the markers obtained by intrusion have 



z coordinates strictly between those of the boundary 
regions. So the xy boundary planes can be obtained by 
taking the extremal z values from among the xy-aligned 
markers. The same process can be repeated for xz- and 
yz-aligned markers to obtain the original box. 

Now consider all markers in the interior of the box. 
The markers fall into six classes determined by the 
direction of their outward facing normal, C, TZ, T, B, 
Q, where the normals for the markers in C point to the 
left, to the right in TZ, to the top in T, to the bottom in 
B, to the front in JF, to the back in Q- Since all interior 
markers come from intrusions, and these intrusions were 
all made in the same face, exactly one of of these classes 
is empty. The boundary face with that class's normal 
is the face in which the intrusions were made. 

To finish the construction, we project all of the 
non-boundary markers that are perpendicular to the 
face onto the face. The result defines a set of non- 
intersecting axis-aligned rectangles. To this set, apply 
Theorem |A.2| to obtain the unique fully consistent set of 
rectangles. Now, just as in the proof of Theorem 5.8 for 
each rectangle, consider the infinite rectangular cylinder 
it defines. If there are no non-boundary markers inside 
that cylinder, then the intrusion goes all the way 
through the box. If there are non-boundary markers 
inside the cylinder, they must all be in the same plane 
since the set of rectangles does not intersect. These 
markers define the depth of the intrusion. 



Proof- [Proof of Theorem 5.10 The proof finds the faces 



of the bounding box in the same way as the proof for 
Theorem |5.9[ Because we no longer have access to the 
normal direction, the method to find the face in which 
the intrusion was made is more involved. 

By hypothesis all of the interior markers must be 
obtained by intrusion from one face. Consider each 
face in turn. Among the interior markers parallel to 
a face, find the marker closest to the face. If there are 
no markers parallel to a face, all intrusions must be from 
this face and must go all the way through the box. If 
there is a marker, check to see if there are perpendicular 
markers closer to the face. If not, this is not the face in 
which the intrusions were made. If so, this is the face, 
because if the closest marker were on the side, rather 
than the bottom, of an intrusion, there would be no 
markers closer to the face than it. 

In this way we have determined the face in which 
all the intrusions have been made. The proof finishes in 
exactly the same way as the proof for Theorem |5.9[ 



